Drop the coreml_static_int8 operators job, which has never once finished - #21695
Merged
Merged
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/21695
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit ad459d8 with merge base 9cd0c12 ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
This PR needs a
|
shoumikhin
force-pushed
the
shoumikhin/drop-coreml-static-int8-operators
branch
from
August 10, 2026 05:51
f3e0024 to
3b0dd40
Compare
shoumikhin
force-pushed
the
shoumikhin/drop-coreml-static-int8-operators
branch
from
August 10, 2026 18:08
3b0dd40 to
ad459d8
Compare
JakeStevens
approved these changes
Aug 11, 2026
JakeStevens
pushed a commit
that referenced
this pull request
Aug 11, 2026
…#21691) ## What is broken Every so often a commit on main shows a cancelled ARM or CoreML job on HUD even though nobody cancelled anything and the commit is fine. It looks like a real failure, so the oncall investigates a phantom. ## Why it is broken GitHub Actions cancels an older run when a newer run lands in the same concurrency group. These workflows use a group key built like this: ``` group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref_name }}-${{ github.ref_type == 'branch' && github.sha }}-${{ github.event_name == 'workflow_dispatch' }} cancel-in-progress: true ``` The key already tells apart pull requests, branches, commits and manual runs. It does not tell apart a scheduled run. These same workflows also have a nightly `schedule` trigger. When the nightly cron fires, `github.ref_name` is `main` and `github.sha` is the current head of main, which are exactly the same values the push run for that commit used. Same key, `cancel-in-progress: true`, so the cron run kills the push run that was still going. The result is a commit whose CI reads as cancelled with no explanation. ## The fix Add `-${{ github.event_name == 'schedule' }}` to the concurrency group, so a scheduled run and a push run on the same commit sit in different groups and no longer cancel each other. This is the same pattern already used by other workflows in this repo. It is applied here to every workflow that has both a `schedule` and a `push` trigger: - `test-backend-arm.yml` - `test-backend-coreml.yml` - `test-backend-openvino.yml` - `test-backend-qnn.yml` - `test-backend-vulkan.yml` - `test-backend-webgpu.yml` - `test-backend-xnnpack.yml` - `test-webgpu-native.yml` - `build-cmsis-pack.yml` - `docker-builds.yml` `doc-build.yml` also has both triggers but is deliberately left alone: it sets `cancel-in-progress: ${{ github.event_name == 'pull_request' }}`, so outside a pull request it never cancels anything and the collision cannot happen there. This only separates schedule from push. It does not change how two scheduled runs of the same workflow behave: `github.sha` is still in the key, so two scheduled runs share a group only while `main` has not moved between them, and that is unchanged by this PR. ## How this was verified Every workflow under `.github/workflows` with a `schedule:` trigger was listed together with its concurrency group, to make sure the change is both correct and complete: - `apple.yml`, `periodic.yml` and `riscv64.yml` already carry exactly the `-${{ github.event_name == 'schedule' }}` suffix this PR adds, so this is not a new idea, it is existing practice in this repo. `periodic.yml` goes one step further and also appends `${{ github.event.schedule }}` to separate its individual crons. - The 10 workflows changed here are precisely the ones still on the older key that also have a `push` trigger, so none were missed. - `build-cadence-runner.yml` and `test-pico2-build.yml` already put `${{ github.event_name }}` in the key, which covers the same collision. - `nightly.yml` has both triggers but only pushes on `ciflow/nightly/*` tags, so its `github.ref_name` is a tag and can never match the scheduled run on `main`. - `doc-build.yml` is the case described above. - The other scheduled workflows have no `concurrency` block at all. Note that the fix only takes effect once it is on `main`, because a scheduled run always uses the workflow file from the default branch. CI on this pull request: 213 checks reported, 195 success, 15 skipped, 2 cancelled, 0 failures. Both cancellations were manual, and in one of them every step had already finished successfully before the cancel landed. ## Overlap with other pull requests Two of the files here are also touched by other open pull requests, in both cases far away from the `concurrency:` block this PR edits: - `.github/workflows/test-webgpu-native.yml` also gets a `timeout:` change in #21690, in the job block further down. - `.github/workflows/test-backend-coreml.yml` also gets a matrix `exclude:` entry in #21695, likewise further down. The hunks do not touch and the changes merge cleanly in any order.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses #21623.
What is broken
Every night, the CoreML backend workflow runs this job on a macOS runner:
It has never finished. It runs for the full 180 minute limit, gets killed, and reports
cancelled. That is about three macOS runner hours per night, every night, for no result at all, on a runner pool that is already the scarcest resource we have.For comparison, in a recent nightly run:
coreml / operatorscoreml_static_int8 / modelscoreml_static_int8 / operatorsSo it is specifically the combination of the heavier quantized flow with the much larger operators suite that does not fit.
Why we cannot just make it faster
The suite already runs with
pytest -n auto, so it is using every core on the runner. There is no sharding support in_test_backend.ymlto split it across several runners. Adding that is real work, and it is the right long term answer, but it is not something to land while the job is burning runner hours nightly.The fix
Stop scheduling that one combination.
_test_backend.ymlbuilds its matrix as everyflowcrossed with everysuite, so today the only way to shape it is to change the flow list, which would also dropcoreml_static_int8 / models, and that one passes and is worth keeping.This adds an optional
excludeinput to_test_backend.ymlthat is wired straight intostrategy.matrix.exclude. It defaults to an empty list, so nothing changes for any other backend.test-backend-coreml.ymlthen uses it to drop the singlecoreml_static_int8plusoperatorspair.After this change the nightly CoreML run is
coreml / models,coreml / operatorsandcoreml_static_int8 / models, all three of which actually complete.When to revert this
When the operators suite can be sharded across runners, or when the quantized flow gets fast enough to fit in the limit. #21623 tracks that.
How this was verified
Two separate things needed checking: that the new input is inert for every other
caller, and that the exclude entry removes the job we mean and nothing else.
_test_backend.ymlhas 8 callers in total. With thisbranch's version of it, the 7 other callers still expand to every flow crossed
with both suites. An
excludelist that matches no combination is simply ignored byGitHub Actions, so the empty default is a no-op.
test-backend-coreml.ymlonly runscoreml_static_int8on thenightly schedule, so a pull request run would not show the effect at all. It
was therefore dispatched on a scratch branch with both flows forced on. The
resulting job list was
(coreml, models),(coreml, operators)and(coreml_static_int8, models). The(coreml_static_int8, operators)pair wasgone and nothing else moved.
Overlap with other pull requests
.github/workflows/test-backend-coreml.ymlis also edited by #21691, which adds aschedule discriminator to the
concurrency:block at the top of the file. This PRadds a matrix
exclude:entry further down. The hunks do not touch and the twomerge cleanly in either order.